Skip to content

refactor: extraction, tooling, and stability#3

Merged
calebevans merged 37 commits into
mainfrom
small-tweaks
Jun 4, 2026
Merged

refactor: extraction, tooling, and stability#3
calebevans merged 37 commits into
mainfrom
small-tweaks

Conversation

@calebevans

Copy link
Copy Markdown
Owner

Summary

  • Overhauls E01 extraction with a TSK-first flow, multi-partition support, and persisted partition offsets for reliable downstream access.
  • Improves investigation quality with stronger validation before confirmation, better behavioral synthesis, and expanded authentication pattern detection.
  • Advances tool architecture with a dynamic executor allowlist, response truncation, composite indexing improvements, and tool_access controls.
  • Adds new forensic tools for file staging detection, timestomping analysis, and autoruns parsing.
  • Stabilizes UI behavior and expands project documentation, while resolving key reliability bugs across gates, case handling, and EVTX persistence.

Key Changes

E01 Disk Image Extraction

  • Reworked extraction to prioritize TSK workflows for disk artifacts.
  • Added robust multi-partition handling and partition discovery resilience.
  • Persisted partition offsets and extraction paths for cross-process continuity.
  • Hardened fallback paths for mount and extraction edge cases.

Investigation Quality

  • Added stronger validation checks before findings are confirmed.
  • Improved behavioral synthesis for clearer cross-artifact reasoning.
  • Expanded authentication pattern detection to better surface suspicious access activity.

Tool Architecture

  • Implemented dynamic executor allowlisting for tighter runtime tool control.
  • Added response truncation and preview handling to keep large outputs manageable.
  • Improved composite tool indexing and orchestration pathways.
  • Introduced tool_access decorator usage for consistent access enforcement.

New Tools

  • Added find_file_staging.
  • Added detect_timestomping.
  • Added parse_autoruns.

UI Stabilization

  • Fixed Rich Live jumpiness and dashboard layout instability.
  • Shifted task panel behavior to a reactive execution model for more stable updates.

Documentation and Reliability Fixes

  • Delivered a broad docs refresh, including architecture and usage coverage plus examples indexing.
  • Fixed gate failure edge cases, case ID lifecycle issues, and EVTX persistence behavior.

calebevans added 30 commits June 3, 2026 01:42
…rhaul

8-agent parallel review identified 78 issues (1 critical, 16 high, 42 medium, 19 low).
All critical, high, and medium issues implemented across 16 specs.

Key improvements:

Pipeline efficiency:
- Replace LLM utility queries with direct tool calls ($2-8/investigation savings)
- Skip redundant extraction when sources already indexed (evidence-path-aware)
- Fix N+1 DB queries, double context builds, sequential subprocess calls
- Parallelize icat extraction with ThreadPoolExecutor
- Reduce prompt token waste (compact results, remove open_case prefix)

Reliability:
- Fix gates auto-passing on None (retry-aware failure tracking)
- Fix race condition in load_case context swap (atomic _ctx replacement)
- Protect populated case DBs from being wiped by second server instance
- Add thread safety (locks) to 4 module-level mutable caches
- Plug resource leaks (FUSE mounts, temp files, pipe deadlock, temp dirs)

Security & correctness:
- Sanitize evidence-derived content before prompt injection
- Accumulate follow-up history across iterations
- Capture batch IDs structurally from tool results instead of regex on prose
- Fix source_is_cited returning True for empty strings
- Add circuit breaker to IOC enrichment, flush audit writes

Code quality:
- Consolidate duplicated IOC/severity/extension/JSON-parsing utilities
- Introduce @tool_access decorator for declarative tool-role mapping
- Introduce @audited_tool decorator for tool boilerplate reduction
- Decompose runner.py (extract compaction loop, token parsing, panel skip)
- Remove dead code (default_registry, orphaned audit prompts)

Investigation quality:
- Add behavioral context synthesis (correlate user activity with suspicious events)
- Add anti-evasion awareness (check for injection within whitelisted JIT/AV)
- Fix finding dedup threshold (merge same-event findings with different wording)
- Exclude false-positive findings from IOC export appendix
- Consolidate registry parser output into one source per hive
- Add document/image forensics guidance to extraction planner

Testing:
- Add 48 new tests (submit_finding, correlator, IOC export, navigator, CLI, extractors)
- Prune 20 low-value tests coupled to private internals
- Final count: 503 tests, all passing

Tool organization:
- @tool_access decorator on all 142 MCP tools with Role enum
- phases.py builds allowlists dynamically from registry (no more manual lists)
- Prompt/tool list contradictions resolved (get_tool_guide, submit_finding, parse_plist)
…access

- Store EVTX extraction dir in DB so index_evtx_file works across server restarts
- Store TSK extraction dirs in DB for same reason
- DB is the single source of truth for all investigation state
… in DB

tool_response now returns status, source, preview (500 chars), and search hint
when a source is provided (indicating data was indexed). Full extraction and
analysis output is only accessible via search/get_raw_output.

Control tools (batch, parallel, case management) and read tools (without a
source) unchanged and return full results. Reduces context pressure on agents
by 10-50x per tool call.

- Refactored hayabusa, hindsight, and mvt to use tool_response consistently
- Volatility batch passes source=None since its per-plugin coordination data
  is not a single indexed source (individual plugins are indexed separately)
- windowed_response (READ path) unaffected
- Executor allowed_tools restricted to plan contents + control essentials
- Reduces executor context from 74 tools to ~15 per session
- Standardize top 30 tool docstrings with what/when/output structure
…on only)

- Remove pre-population from planner output (unreliable, causes stale entries)
- Tasks appear only when actually dispatched or called
- Clear task panel on phase retry to prevent accumulation
- Consistent real-time updates from job completion markers
- Remove SleuthKitExtractor class (extractors/sleuthkit.py)
- Remove BulkExtractorExtractor class (extractors/bulk.py)
- Remove PlasoExtractor class (extractors/plaso.py)
- Remove LogFileExtractor class (extractors/logs.py)
- Remove EZToolsExtractor class (extractors/eztools.py)
- Remove DiskImageExtractor class and its private helpers from disk.py
- Remove VolatilityExtractor class from volatility.py
- Remove Extractor protocol and ExtractorRegistry (extractors/base.py)
- Remove ExtractionResult dataclass (unused by server or orchestrator)
- Remove tests/test_extractors.py (tested only dead extractor classes)

Retained from disk.py: _parse_evtx_file, _mount_image, _unmount_image
(used by server/tools/extract/evtx.py and server/extract_helpers.py).

Retained from volatility.py: _find_vol_binary, _plugin_short_name
(used by server/tools/extract/volatility.py and server/tools/yara.py).

Retained: extractors/classifier.py (EvidenceClassifier used by MCP tools).
- Query enrichment.iocs source for threat intel metadata
- Add context column to IOC tables showing country, ASN, abuse score
- Works in both markdown and HTML reports
…tate

- Update CLI usage with required case_id argument
- Document tool_access decorator pattern for contributors
- Update architecture with dynamic allowlists, composite indexing, gate changes
- Add investigation quality improvements section
- Step-by-step walkthrough from function creation to registration
- Covers tool_access, indexing, skip logic, and testing
- Linked from README
…nsaw

- run_hayabusa falls back to DB when in-memory EVTX path unavailable
- run_chainsaw gets same treatment if affected
- Consistent with index_evtx_file fix from earlier
- Match system names against relative path instead of filename only in
  _build_evidence_context, so files in nested subdirectories (e.g.
  /evidence/system-10.3.58.4/system-disk/system.E01) are discovered
  even when the system name only appears in a parent directory
- Intersect dynamic executor allowlist with role-based permissions,
  preventing the planner from granting the executor access to tools
  outside its declared role (e.g. scan_evidence leaking from catalog)
- Enforce MULDER_CASE_ID in create_case, refusing to create per-system
  databases when an orchestrator-level case ID is set
- Enforce MULDER_CASE_ID in scan_evidence over any explicit case_id
  argument, preventing agents from bypassing the enforced case
- Fix test mocks for _run_analyst task_system parameter
mmls consistently exits with code 1 and empty stderr on images that
lack a partition table (partition dumps, single-filesystem images).
The previous error response was generic and gave no recovery guidance.

Introduce _classify_mmls_failure to categorize mmls failures into
three types: no_partition_table (empty stderr), ewf_unsupported
(E01/libewf keywords in stderr), and generic mmls_failed. Each type
returns a structured error_type and actionable suggestion directing
the agent to use run_fls with partition_offset=0.
…allel

_handle_job_completion broadcast each JOB_COMPLETE event to every active
system, so when system A's run_fls finished it also marked system B's
run_fls as done. Add complete_one_running_task to the dashboard, which
updates exactly one task in "running" state per completion event, and
switch the log tailer to use it instead of the broadcast loop.
- Equal task lines per system regardless of task count
- Fix log panel to fixed height to prevent layout jumps
- Show running tasks first, cap done tasks per system
…refreshes

- Always render two-column layout (never switch between 1 and 2 columns)
- Empty task panel when no tasks instead of removing the column
- All panels have fixed heights that never change between refreshes
The Rich Console was initialized with stderr=True, tying its output to
fd 2.  Any stray write to fd 2 from the Claude Agent SDK subprocess or
native code would land at the cursor position below the Live panel,
persisting for one refresh cycle (~0.5s) before the next redraw
overwrote it.

Changes:
- Duplicate fd 2 at init so the Console always reaches the terminal
  through a private fd, independent of the process-wide stderr.
- On start(), redirect fd 1 and fd 2 to /dev/null; restore on stop().
  This silences ALL subprocess and native-code writes at the OS level.
- Remove the dead _refresh() method (Live.update sets _renderable which
  get_renderable never reads; the auto-refresh callback was always the
  sole render path).
- Increase refresh_per_second from 2 to 4 for smoother visual updates.
- Fix pre-existing SIM102 lint errors in runner.py and yara.py.
finalize_report reads model_usage.json for real token counts instead of
falling back to audit log estimates. The sidecar file is now written
before the report phase starts so it exists at render time.
- Removed outdated image from README and consolidated documentation sections.
- Introduced a new glossary file to define project terminology and concepts for contributors and users.
- Updated README to include links to comprehensive documentation resources, including architecture, tool addition guides, and a tool manifest.
- Added `case_id` argument to the `investigate` command for better tracking of investigations.
- Updated documentation to clarify the use of `case_id` in CLI commands.
- Improved terminal size handling in the dashboard to prevent errors when retrieving terminal dimensions.
- Adjusted prompt templates to include case ID instructions for clarity.
- Refactored tests for better readability and consistency.
- Correct CLI option docs (export-iocs, export-navigator, report)
- Fix architecture claims (retry scope, throttling, PDF, exports, enrichment)
- Fix adding-tools guide (variable names, pattern conflicts, verification steps)
- Correct tool-manifest types and role tags
- Fix glossary source references
…t fails

When run_fls succeeds with an explicit partition offset but mounting
fails (multi-segment E01s, missing FUSE, container restrictions), the
partition offset was lost. Downstream tools (registry, EVTX, Amcache,
ShimCache, MFT, Prefetch) that fell back to icat extraction would use
offset 0, causing all extractions to fail.

- Store partition offset in DB kv_store on successful run_fls
- Add _resolve_partition_offset helper (kv_store -> tsk.partitions -> mmls)
- Shared helper for cross-tool offset reuse in _tsk_extract_files,
  _extract_evtx_from_image, _run_fls_inline, and run_mft_parser
…offset failure

- Fix mmls output regex to handle standard TSK 4.x format with spaced
  slot fields (e.g. "002:  000:000") and optional leading whitespace;
  the previous regex only matched compact "002:000" format, silently
  returning offset 0 even when mmls found a valid partition table
- Restructure run_fls to be self-contained: check kv_store first, try
  offset 0, and on failure run mmls inline to detect and retry with the
  correct offset, eliminating race condition with parallel run_mmls
- Consolidate _detect_partition_offset to delegate to _parse_partition_offset,
  adding linux and "basic data" (GPT) partition type detection
Reverse the extraction strategy for registry, prefetch, amcache,
shimcache, and MFT parsers: attempt TSK extraction (fls + icat) first
since it reads E01 images directly without FUSE, then fall back to
mount-based extraction as a bonus path. This eliminates the container
dependency on ewfmount/guestmount which require privileged mode.
- Defer temp dir creation in _tsk_extract_files until first pattern
  match to prevent leaked empty directories when fls data exists but
  no files match
- Log icat timeout and failure in run_mft_parser so fallback to mount
  is observable in logs
- Update docstrings to reflect TSK-first extraction order
run_fls now analyzes all non-trivial partitions, not just the largest.
Each partition's files indexed separately with partition-aware source
names (tsk.filelist for primary, tsk.filelist.p1/p2/etc. for secondary).
Primary partition offset stored in kv_store for downstream tools;
secondary partitions accessible via search on their source names.

Downstream consumers (_tsk_extract_files, _extract_evtx_from_image,
_find_inodes_by_pattern, list_files, get_deleted_files) updated to
search all tsk.filelist* sources and use the correct per-partition
offset for icat extraction.
…tence, and staging

Add three methodology improvements to analyst prompts based on gap analysis:
- Authentication pattern analysis: guide analysts to examine failure/success
  clusters and verify source addresses before dismissing logon events as normal
- Persistence coverage: ensure all relevant log sources (including system-level
  logs) are indexed before concluding no persistence was found
- Data staging detection: search filesystem metadata for archive files in
  staging locations, not just execution artifacts for archiving tools
…stomp detection

- find_file_staging searches MFT/filelist for archive creation and deletion patterns
- index_evtx_file auto-indexes System.evtx alongside Security for persistence coverage
- detect_timestomping identifies SI vs FN timestamp mismatches in MFT data
- Add multi-partition, TSK-first extraction, new tools to architecture
- Add find_file_staging and detect_timestomping to tool manifest
- Update glossary with new terms
- Verify README accuracy
- Indexes Autoruns CSV as searchable source for persistence detection
- Auto-discovers autoruns CSV files in evidence directories
- Updated tool manifest, glossary, and source prefix registry
@calebevans calebevans self-assigned this Jun 4, 2026
@calebevans
calebevans merged commit fefca55 into main Jun 4, 2026
6 checks passed
@calebevans
calebevans deleted the small-tweaks branch June 5, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant